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The Race to the Top—Early Learning Challenge (RTT-ELC) grants program, sponsored by 
the U.S. Department of Education (ED) and U.S. Department of Health and Human Services 
(HHS), aimed to increase the number of high quality early learning and development programs. 
RTT-ELC awarded $520 million in the first of three rounds of grants to help states develop and 
implement systems that rate early learning and development programs based on state-defined 
quality standards and help them improve on these standards. These systems are known as tiered 
quality rating and improvement systems (TORIS). Programs at the lowest level meet a basic level 
of quality, whereas programs at the top levels meet the highest standards of quality as measured 
by one of these rating systems. 


A key objective for RTT-ELC was to increase the number of early learning and development 
programs in the top rating levels of the TORIS, and to increase the number of children with high 
needs, such as those from low-income households, enrolled in those programs. To meet this 
objective, states could try to encourage more programs to participate in TORIS and achieve high 
ratings. 


The likelihood that programs participate in TORIS and achieve top rating levels could be 
influenced by the system characteristics and policies that states have selected for their TORIS. 
These characteristics and policies could also affect the accuracy of the ratings as measures of 
programs’ quality. However, little is known about how states’ actual implementation of TORIS 
characteristics and policies relate to programs’ ratings. Understanding the patterns of ratings 
across states with different characteristics and policies could help inform next steps in the 
continued testing and development of TORIS. These actions could ultimately help states advance 
toward the goal of increasing children’s access to high quality programs. 


This brief describes programs’ ratings across the nine Round I RTT-ELC states, which had 
different TORIS characteristics and policies in place during the grant period (2012 to 2016). Over 
this period, individual states changed few of their TORIS characteristics and policies. TORIS 
characteristics could affect all programs within a state, but the policies were often directed at 
particular types of programs. Thus, this brief presents patterns for several key TORIS 
characteristics and policies, individual states, and four types of programs: (1) state-funded 
prekindergarten (pre-K) programs, (2) Head Start programs, (3) licensed centers that received 
child care subsidies, and (4) licensed centers that did not receive subsidies. The first three types 
of programs—state-funded pre-K, Head Start, and licensed centers that received subsidies—serve 
low-income children; the last type might serve fewer low-income children because they do not 
receive public funding to do so. The TORIS characteristics and policies examined include the 
following: 


e Three policies that could affect the accuracy of programs’ ratings at entry: alternative 
pathways, automatic ratings, and entry level requirements. Alternative pathways and 
automatic ratings award certain types of programs (such as state-funded pre-K and Head 
Start programs) credit for meeting all or part of the TORIS standards. These policies are 
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based on the theory that these programs would likely qualify for high ratings because 
they meet external quality standards (such as those required to receive state pre-K and 
Head Start funds). Conversely, requirements that certain types of programs (such as 
licensed centers) enter at the lowest rating level could result in programs receiving lower 
ratings than they otherwise would have qualified for. Similar patterns of ratings for 
programs in states with and without these rating level policies might imply that the 
policies do not markedly reduce the accuracy of the ratings. 


Two characteristics and one policy that could affect programs’ attainment of top rating 
levels: the length of time TQRIS had been implemented, the rating structure that 
determines programs’ ratings, and financial incentives that increase as programs 
reach higher ratings. Longer-running TORIS might have more programs at top rating 
levels because programs had additional time to improve. Rating structures that require 
programs to meet all standards of a given rating (building block) may hold programs at 
lower rating levels than structures (points or hybrid) that give programs the flexibility to 
earn a rating by accumulating enough points across different components of the 
standards. A financial incentive policy (tiered subsidy reimbursement rates) that 
increases programs’ child care subsidies when they reach higher rating levels may 
encourage programs that serve children who receive these subsidies to achieve higher 
ratings. 


Based on administrative and interview data from the nine states, programs’ ratings at entry 
and attainment of top rating levels varied substantially across states, potentially because of the 
characteristics and policies examined, as well as other differences across states. Key findings 
include the following: 


State-funded pre-K and Head Start programs generally had high ratings at entry. These 
programs did not consistently enter at higher ratings in states that offered alternative 
pathways or automatic ratings than in states that did not. 


Licensed centers that did and did not receive subsidies generally had low ratings at 
entry. Only licensed centers that did not receive subsidies were more likely to enter at the 
lowest rating level in states that required them to do so than in states that did not. 


States that had implemented statewide TORIS for 10 years or more by 2016 had higher 
percentages of programs at the top two rating levels than those with fewer years of 
implementation. 


Substantially lower percentages of programs achieved the top two rating levels in states 
with building block rating structures than in states with hybrid or points rating 
Structures. 


Tiered reimbursement rates were associated with higher percentages of licensed centers 
that received subsidies achieving top rating levels by 2016, but states with tiered 
reimbursement rates had higher ratings for all types of programs, even those that did not 
receive this financial incentive. 


These findings suggest several issues that could be investigated by future research. For 


example, future research could examine how each of these policies and characteristics affects 
programs’ quality, using consistent measures of the quality of the child care environment or 
children’s development across different states and types of programs. 
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Providing young children from birth to age 5 with high quality early learning and development 
experiences, particularly in formal center-based and preschool settings, yields significant 
benefits—especially for children from low-income and disadvantaged households (Dearing et al. 
2009; Weiland and Yoshikawa 2013). These benefits can include short-term improvements in 
social skills, behavior, and language skills, as well as long-term effects, such as pursuing more 
years of education and earning higher income (Campbell et al. 2002; Heckman 2011; Duncan and 
Manuson 2013). To improve the quality of early learning and development that children 
experience, states created TQRIS that establish quality standards, encourage programs to 
participate, and rate programs on the standards (Zellman et al. 2011). These systems also provide 
incentives for programs to earn higher ratings on these standards and publicize the ratings of 
individual programs to help parents choose higher quality programs for their children. 


Developing and implementing TQRIS was a key 
aspect of the RTT-ELC program. Through TQRIS, 
RTT-ELC sought to strengthen the quality of early | + Developing and adopting a common, 
learning and development programs by promoting ier eugene 
progress on five objectives (Box 1), including | 2- Promoting participation in the TQRIS 
increasing access to high quality programs, particularly | 3. Rating and monitoring early learning 


Box 1. RTT-ELC’s five TQRIS objectives 


for children from low-income and disadvantaged and development programs 
households (U.S. Government Printing Office 2011). _ Promoting access to high quality 

The Departments of Education (ED) and Health and programs for children with high needs 
Human Services awarded RTT-ELC grants through by: 

three rounds of competition. Round 1| grants, which this e Increasing the number of programs 
brief focuses on, were awarded in December 2011 to in the top levels of the TQRIS, and 
nine = states—California, Delaware, Maryland, Increasing the number and 
Massachusetts, Minnesota, North Carolina, Ohio, percentage of children with high 
Rhode Island, and Washington. Of the Round | states, needs who are enrolled in programs 
five had statewide TQRIS by 2011, the year in which that are in the top levels 
RTT-ELC grants were awarded. By 2015, eight of the . Validating the effectiveness of the 
states had statewide TQRIS, and the ninth state— TQRIS 


California—had locally administered TQRIS_ but 
statewide adoption was underway. 


The Institute of Education Sciences (IES) at ED initiated a study to learn about TQRIS in the 
nine states that received RTT-ELC Round | grants and to inform ongoing TQRIS development. 
The study focused on center-based early learning and development programs that served 
preschool-age children (these programs might have also served infants, toddlers, and school-age 
children). The study generated a series of reports and briefs. 


The first report examined progress on the first three TQRIS objectives (outlined in Box 1) by 
describing the development, structure, and characteristics of TQRIS in the Round | states. It found 
that these states made progress developing statewide TQRIS (Kirby et al. 2017). However, they 
varied substantially in the ways they promoted participation in TQRIS, defined quality standards, 
verified and monitored that programs met the standards, and calculated ratings. Future work from 
this study plans to examine states’ progress on the fourth and fifth TQRIS objectives, describing 
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(1) TQRIS participation and ratings across the Round | RTT-ELC states, specifically for types of 
programs that serve low-income children, and (2) findings from states’ TQRIS validation studies. 


This brief contributes to the larger study by assessing part of the fourth objective—promoting 
access to high quality programs by increasing the number of programs at top levels of the TQRIS. 
It describes patterns of ratings across the nine Round | RTT-ELC states, which had different 
TQRIS characteristics and policies in place during the grant period (2012 to 2016). These 
characteristics and policies could influence the level at which programs enter the TQRIS and 
whether they attain the highest rating levels. Therefore, it is useful to understand how states with 
different TQRIS characteristics and policies compare in their programs’ attainment of the top 
rating levels. 


This brief also contributes to the larger body of research on TQRIS in two key ways. First, it 
uses data on ratings from a large sample of programs across nine states. Thus, it can compare 
ratings for the same program types across states, as well as for different program types within 
states. Second, the brief describes programs’ ratings under realistic implementation of the TQRIS 
characteristics and policies. This differs from simulation studies, which can describe how 
programs’ ratings might theoretically change under different characteristics or policies (such as a 
different rating structure) but do not examine actual ratings (Isner et al. 2012; Faria et al. 2015). 
Thus, this brief complements those studies by providing evidence about aspects of implementation 
that could result in different ratings than might be theorized. 


Research Questions 


To understand patterns in state-defined TQRIS ratings across RTT-ELC states with different 
TQRIS characteristics and policies, we examined two research questions: 


1. What were the patterns in rating levels at TQRIS entry across states with different rating 
level policies? 


2. What were the patterns in programs’ attainment of the top two rating levels by 2016 across 
states with different TQRIS characteristics and policies? 


For both of these research questions, we examined how TQRIS ratings varied across states. 
In addition, for the TQRIS policies, we presented patterns across states for each type of program 
that they targeted. To provide context for each of the research questions, we also present patterns 
of ratings at TQRIS entry and by 2016 for all states. 


The answers to these research questions help shed light on how states’ TQRIS characteristics 
and policies might relate to their progress in increasing the number of programs at top rating levels. 
First, if the same program types enter at different rating levels in states with rating level policies 
and states without them, this might raise questions about whether the policies artificially push 
programs toward particular ratings. Such a finding could also suggest that these policies might 
reduce the accuracy of ratings as measures of the programs’ quality. Second, differences between 
programs’ attainment of the top two rating levels across states with different TQRIS characteristics 
and policies could suggest that these characteristics and policies might contribute, in part, to these 
differences. Other differences across states could also be responsible for the findings, but the 
patterns could help identify topics for discussion and future investigation. 
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In Boxes 2 and 3, we describe the data sources, methods, and samples we used to answer these 
questions. 


Box 2. Data sources 


Telephone interviews with TQRIS administrators in the nine states provided information about TQRIS 
characteristics from 2012 through 2016. We conducted two rounds of interviews. The first round, from 
October 2014 through April 2015, collected information on TQRIS characteristics from the start of the RTT- 
ELC grants in 2012 through the time of the interview. The second round, in December 2016 and January 
2017, confirmed or updated this information. 


Administrative data from the nine states provided information about TQRIS ratings from the start of the 
RTT-ELC grants in 2012 through June 2016. Data for 2016 are for only half the year (from January through 
June). Maryland did not officially launch its TQRIS until 2013, so its data span 2013 to 2016. Data from 
California are from the 16 counties in which the state implemented its TQRIS beginning in 2012. California 
provided data for these counties for 2014 to 2016. All other states have statewide TQRIS systems and we 
collected data for each of those states as a whole. In Ohio, data from before and after October 2013 (when 
the state moved from a system with three rating levels to one with five levels) were not comparable. For 
these reasons, we only used data from Ohio and California from 2014 to 2016. 


We combined the TQRIS data with other administrative data, including data on licensing and child care 
subsidies, and state-funded pre-K and Head Start program lists and data, to build a longitudinal picture of 
early learning and development programs over the study period. All states provided data on state-funded 
pre-K, Head Start, and licensed programs that did and did not participate in TQRIS. However, most states 
did not provide data on license-exempt programs that were not either state-funded pre-K or Head Start 
programs (which are often considered license-exempt when operated in public schools), particularly if they 
did not participate in TQRIS. Only three states allowed license-exempt programs that were not receiving 
state pre-K or Head Start funds to participate in TQRIS. Such programs could include those administered 
on Tribal lands or on military bases, or those affiliated with a religious program. These programs accounted 
for less than 1 percent of TQRIS participating programs in 2016 in two of the three states and less than 5 
percent in the third state. 


For more information about the data sources, see Appendix A. 
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Box 3. Methods and samples 


We used descriptive analyses to describe programs’ ratings overall and by key TQRIS characteristics and policies, state, 
and type of program. For these analyses, we used standardized rating categories to describe ratings, calculated averages 
across groups of states, and described programs’ ratings at entry and attainment of top rating levels by 2016 (the end 
of the RTT-ELC grant period). 


The analyses focus on four types of programs. Due to data limitations, all analyses exclude license-exempt programs 
that were not either state-funded pre-K or Head Start programs (which are often considered license-exempt when 
operated in public schools). 


Standardized rating categories 


We used five standardized rating categories that describe the position of rating levels within each state’s rating structure. 
Seven of the nine states used a five-level rating structure and two states used four levels. Rating levels were not directly 
comparable across states, even among states with the same number of ratings, because states included different quality 
components and calculated the ratings differently (Kirby et al. 2017). The four-level rating structure only uses four of the 
categories: in this structure, 1 is the lowest level, 4 is the highest level, and 2 and 3 are the second lowest and second 
highest levels, respectively. There is no middle level in the four-level rating structure, because the exact middle of 1 and 
4 is 2.5. 


The standardized categories align the top two rating levels across the four- and five-level structures, following the most 
common and conservative definition of “top tiers” that RTT-ELC states used. In their grant reports, the majority of RTT- 
ELC states only included ratings in the top two levels (4 and 5 in the five-level structure and 3 and 4 in the four-level 
structure) in their “top tiers.” These categories standardized the ratings as shown in the table below. 


Rating in four-level Rating in five-level 
Standardized rating level category rating system rating system 
Highest level 
Second highest level 
Middle level 
Second lowest level 
Lowest level 
n.a. = not applicable. 


The analysis examined the percentage of programs that received each rating level category and the percentages in the 
two highest levels combined. 


Averages across states 


We present (1) averages for all states, states that had each characteristic or policy, and states that did not have this 
characteristic or policy, and (2) patterns for individual states. Because each state represents a separate implementation 
of a characteristic or policy, each state received the same weight in the averages. Supplemental analyses also calculate 
the median ratings for (1) programs within states and (2) all states, based on an average distribution where each state 
received the same weight. 


For these analyses, we used the full population of programs, as opposed to a sample. Thus, any observed differences 
reflect actual differences, not estimates based on statistical tests. 


Ratings at entry 


To answer the first research question, we examined programs’ ratings at entry. This analysis focuses on the 9,658 
programs that entered TQRIS during the analysis period (2012 to 2016 for seven states and 2014 to 2016 for California 
and Ohio). 


Attainment of top ratings 


To answer the second research question, we examined programs’ attainment of top rating levels by 2016. To understand 
how programs reached these two highest levels, we examined whether they received these ratings at entry or through 
movement to highest rating levels. For programs that reached the highest levels before the analysis period began (2012 
for seven states and 2014 for California and Ohio), we did not observe how they reached these levels (at entry or through 
movement). This analysis included 13,357 programs. Although this analysis focuses on programs’ attainment of top 
rating levels, Appendix B shows the full distribution of programs’ ratings by 2016 for each TQRIS characteristic or policy. 
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Types of Participating Programs and TQRIS Policies 


Understanding the different types of programs that participate in TQRIS and _ the 
characteristics and policies that states can select for the systems is essential for exploring patterns 
of ratings. Programs serve children of various age ranges from different income groups, and they 
differ on other characteristics that could affect their ratings. States’ TQRIS characteristics may 
influence how all programs enter and move through the system. TQRIS policies often target certain 
types of programs to encourage them (especially those that serve low-income children) to 
participate in the rating system and earn higher ratings. Below we describe the program types, 
TQRIS characteristics, and TQRIS policies for the nine states. This brief focuses on just a few 
characteristics and policies, but understanding a more complete picture is necessary context for 
interpreting the patterns in ratings. 


Program types. The landscape of center-based early learning and development programs that 
serve children ages 3 to 5 is quite diverse. Programs differ in the ages of children they serve and 
the degree to which they serve low-income or high-needs children (Table 1). State-funded pre-K 
and Head Start programs specifically target preschool-age children and fully, or predominantly, 
serve low-income children. Licensed centers can serve children from birth through age 12; in this 
study, we included only those that served preschool-age children (these programs might have also 
served infants, toddlers, and school-age children). Licensed centers that received child care 
subsidies served at least some low-income children; we could not determine from the data whether 
licensed centers that did not receive subsidies served low-income children. Other differences 
among programs include the type of setting (such as within a public school or a community-based 
organization), the type and source of their licensing or regulation, and the types of funding or 
revenue they receive. 


Many programs receive funding from a mix of sources, so categorizing each program as a 
single type might not fully capture its nature. For example, a community-based center could have 
Head Start classrooms for children ages 3 to 5 and state-funded pre-K classrooms for 4-year-olds. 
It could also receive Child Care and Development Fund subsidies to support low-income children 
from birth to age 12 and receive full or partial tuition directly from parents. However, to explore 
patterns across programs and how TQRIS characteristics might influence them, we used 
assignment rules to create mutually exclusive program types (see Appendix A for details about 
program type assignments and Table A.1 for information on their licensing status and funding 
sources). We categorized programs into four types to mirror those typically found in the state RTT- 
ELC yearly reports: (1) state-funded pre-K programs, (2) Head Start programs, (3) licensed centers 
that received child care subsidy funding, and (4) licensed centers that did not receive any type of 
public funding. 


Not all Head Start and state-funded pre-K programs must hold a license from the state child 
care licensing entity. However, these programs must follow standards set by the Federal Office of 
Head Start or state department of education, respectively. These standards include components of 
quality that are similar to those of TQRIS standards and often set requirements that are comparable 
to those for the highest TQRIS rating levels. For example, state-funded pre-K programs in the nine 
states require the use of early learning development standards and approved curricula (Barnett et 
al. 2017). Therefore, Head Start and state-funded pre-K programs could likely receive high ratings, 
even without policies that grant them credit for all or part of the TQRIS standards. 
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Table 1. Types of early learning and development programs in the nine Round 1 RTT-ELC states 


Type of program Children served Setting Licensing or regulation Public funding received 
State-funded All states serve 4-year- Community-based Regulated by state State pre-K funds 
pre-K olds; 5 states also serve or public school departments of education 
3-year-olds settings using state program May also have received 
standards Federal Head Start funds and 
Targeted to low-income child care subsidies 
children in 7 states May be licensed in 
community-based settings or 
considered license-exempt 
when delivered in public 
school settings 
Head Start Targeted to low-income Community-based Regulated by the Federal Federal Head Start funds. 
children ages 3 to 5 or public school Office of Head Start using 
settings Head Start Program May also have received state- 
Performance Standards pre-K funds and child care 
subsidies 
May be licensed in 
community-based settings or 
considered license-exempt 
when delivered in public 
school settings 
Licensed Range from birth to Community-based — Licensed by child care Federal Child Care and 
centers, school-age (up to age 12)? _ settings licensing entity in the state Development Fund 
received 
subsidies Subsidies support low- State-funded child care 
income children; may also subsidies 
serve children from 
higher-income levels 
Licensed Range from birth to Community-based — Licensed by child care None> 
centers, no school-age (up to age 12) settings licensing entity in the state 
subsidies 


May serve low-income 
children but data 
available on public 
funding sources do not 
indicate this 


Source: Document reviews and telephone interviews conducted from December 2016 to January 2017. 

aThe analyses included only licensed centers that served preschool-age children. 

>Centers did not receive the public funding examined in this analysis including child care subsidies, Head Start, or state pre-K funds. 
RTT-ELC = Race to the Top-Early Learning Challenge. 


TQRIS characteristics and policies. TQRIS first emerged in the late 1990s, expanded 
somewhat in the early 2000s, and flourished under RTT-ELC. By the end of 2017, 44 states had 
TQRIS (BUILD Initiative 2017).! TQRIS rate early learning and development programs against 
state-defined quality standards that include components such as licensing compliance, quality of 
the learning environment, and qualifications of the workforce. Based on meeting the standards, 
programs receive an overall rating level intended to document variations in the quality of care that 
children receive. In the Round | RTT-ELC states, the overall rating ranged from 1 to 4 in two 
states and 1 to 5 in seven states. 


The meaning of each rating level—what it measures and how—is not the same across states. 
States define TQRIS standards and rating levels differently (for more information about how the 
nine Round 1 RTT-ELC states defined rating levels, see the first report from this study, Kirby et 
al. 2017). For example, the nine Round 1 RTT-ELC states use anywhere from 6 to 12 components 
for their highest rating level. Even when multiple states use a particular TQRIS component (such 
as the quality of the learning environment), each state might differ in the rating level it assigns to 
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programs with that component. The commonality across the states is that the higher the rating, the 
higher the level of quality of the program based on the state’s TQRIS standards. 


States also differ in the combinations of characteristics and policies that they define for their 
systems, which could influence programs’ participation, the ratings they receive at entry, and 
whether they attained the top levels by 2016. The nine Round 1 RTT-ELC states varied widely in 
how they chose to structure their TQRIS. These nine states are not representative of all TQRIS, 
but they do reflect the wide variation in characteristics and policies that exists across the 44 current 
TQRIS (BUILD Initiative 2017).? 


TQRIS characteristics. We focus on two characteristics of TQRIS that could influence the 
frequency and ease with which programs can achieve higher ratings (Table 2): 


Length of implementation. A state TQRIS needs about two to four years, on average, to 
achieve a steady state of implementation in which the policies, practices, and supports are in 
place and running smoothly (Metz and Bartley 2012). Mature systems might have more- 
established policies that aim to encourage and support movement through the ratings. At the 
beginning of the study period in 2012, four states had implemented TQRIS statewide for two 
years or more. North Carolina had the longest-running TQRIS, which it adopted in 1999. 
Other states expanded or launched TQRIS closer to the start of the RTT-ELC grants. Two of 
the nine states expanded TQRIS from a pilot to statewide at the start of RTT-ELC in 2012 
and two others first launched TQRIS concurrently with RTT-ELC. 


Rating structure. TQRIS use one of three structures to determine a program’s rating level: 
building block, points, and hybrid structures (Figure 1). Six states use points and hybrid 
structures, which can provide programs with flexibility in demonstrating quality because 
they offer multiple ways to earn a higher rating. These structures might promote more 
movement of programs up the rating levels than building block structures, which require 
programs to meet all standards of a given rating. 


Table 2. TQRIS characteristics in RTT-ELC Round 1 states 


Characteristics included in analysis Additional characteristic for context 


Length of TQRIS implementation 
State in 2012 Rating structure@ Validity period for ratings 


California First year Points 2 years 
Delaware 4 years Hybrid 1-3 years by rating level 
Maryland Did not start until 2013 Block 1 year 
Massachusetts 1 year Block 2 years 
Minnesota First year Hybrid 2 years 
North Carolina 13 years Points 3 years 
Ohio 6 years Hybrid 1-3 years by rating level 
Rhode Island 3 years Block 3 years 
Washington First year Hybrid 3 years 


Source: Telephone interviews conducted by Mathematica, October 2014-April 2015 and December 2016-January 2017. 


a Rating structure applies to levels beyond the lowest level. All TQRIS have a block structure for the lowest level that requires programs to 
meet all licensing requirements and sometimes additional components. 
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Figure 1. Rating structures in TQRIS 


Building block Points Hybrid 
v ee i. a 
a. OC re 


Ua 


Programs must meet all Programs earn points for Programs are rated by a 
standards in a level to meeting standards. The total building-block structure in 
receive that rating. number of points determines some levels (usually, lower 
the rating. levels} and a points structure in 


others (usually, higher ones}. 


To provide additional context for the findings, we discuss one other characteristic of the 


systems that may also affect movement: the validity period for ratings. Depending on the state 
(and sometimes the rating level), ratings are valid from one to three years, and then programs must 
reapply for a rating. States’ decisions about how often programs must reapply for a rating could 
contribute to the frequency of changes in ratings. 


TQRIS policies. The nine states developed policies to attempt to encourage TQRIS 


participation among different types of programs and support ratings improvement (Table 3). This 
brief focuses on four policies: 


Alternative pathways or automatic ratings. Six states aimed to encourage participation by 
awarding credit (known as offering an alternative pathway) for certain quality components 
(five states) or automatically awarding higher rating levels (one state) to programs that met 
quality standards external to the TQRIS (for example, state-funded pre-K and Head Start 
programs). These policies exempt eligible programs from part or all of the TQRIS rating 
process in an effort to decrease the burden on programs that must meet multiple sets of 
standards. Some states have these policies for one type of program but not others. For 
example, Massachusetts offers alternative pathways to Head Start programs but not state- 
funded pre-K programs. 


These policies could result in programs entering the TQRIS at higher rating levels than they 
would have qualified for based on the TQRIS standards. When developing these types of 
policies, states often complete a crosswalk between other external quality standards and the 
TQRIS standards to establish comparability. However, the specific degree of equivalence 
between the relevant external standards and TQRIS standards is not fully known; 
equivalence can more readily be established for specific components used in alternative 
pathways than for a full set of components that automatically qualifies a program for a high 
rating level. 
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Table 3. TQRIS policies in RTT-ELC Round 1 states 


Policies included in analysis 


Additional policies for context 


Tiered 
Entry reimbursement 
requirement at payments for Rating level 
Alternative Automatic lowest rating children receiving Participation funding 


pathways 


rating 


level 


subsidies 


requirements 


requirement 


California None None None None None None 
Delaware None Head Start, Licensed centers Any program State-funded pre- None 
state-funded that received receiving subsidies K 
pre-K subsidies for low-income 
Licensed centers children 
with no subsidies 
Maryland State-funded None None Any program Licensed centers  State-funded pre- 
pre-K (school- receiving subsidies _ receiving Ka 
based only) for low-income subsidies, 
children state-funded pre- 
K 
Massachusetts Head Start None Licensed centers None Licensed centers  State-funded pre- 
that received receiving K 
subsidies subsidies, 
Licensed centers Head Start, 
with no subsidies state-funded pre- 
K 
Minnesota Head Start, None None Any program None None 
state-funded receiving subsidies 
pre-K for low-income 
children 
North Carolina None None All participating Any program All licensed Licensed centers 
programs receiving subsidies programs; receiving 
for low-income includes both subsidies, 
children types of licensed state-funded pre- 
centers and may K 
include Head 
Start and state- 
funded pre-K 
programs in 
licensed settings 
Ohio None None None Any program Licensed centers None 
receiving subsidies _ receiving 
for low-income subsidies, state- 
children funded pre-K 
Rhode Island Head Start None None Any program Licensed centers None 
receiving subsidies _ receiving 
for low-income subsidies, 
children state-funded pre- 
K 
Washington Head Start, None Licensed centers Any program State-funded pre- None 
state-funded that received receiving subsidies K 
pre-K subsidies for low-income 


Licensed centers 
with no subsidies 


children 


Source: Telephone interviews conducted by Mathematica, October 2014-April 2015 and December 2016-January 2017. 


Note: 


a These policies took effect in the middle of 2015 in Maryland. 


Cells that say “None” indicate states that did not have the policy. All other cells are shaded. 


e Tiered reimbursement for child care subsidies. Seven states offered higher payments to 
subsidize serving low-income children if the programs participated in TQRIS. The payments 
also increased with each rating level.* The tiered rates might also have promoted programs’ 
advancement, particularly among licensed centers that received subsidies. 
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To provide additional context for the findings, we discuss two additional policies that could 


have affected programs’ participation and ratings: 


Participation requirements. Seven states required certain types of programs (for example, 
those that received public funding like state-funded pre-K programs, Head Start programs, 
or licensed centers that received child care subsidies) to participate in the TQRIS. 
Participation of licensed centers that did not receive public funding was voluntary in every 
state but North Carolina, which required all licensed programs to participate in the TQRIS. 


Rating level funding requirement. In three states, programs had to attain a certain rating 
level to be eligible for specific types of funding. For example, a program in North Carolina 
must have attained a level 3 rating to receive child care subsidies from the Child Care and 
Development Fund or a level 4 rating to receive state pre-K funds. These policies could have 
also incentivized particular programs to attain certain ratings at TQRIS entry. 


In addition to these policies, states could try to promote quality improvement by publicizing 


TQRIS ratings. This study did not collect systematic data on how states publicized ratings that 
could be used to interpret the patterns of ratings at entry and attainment of the highest rating levels. 


Below we discuss the patterns in TQRIS participation and ratings among programs in the nine 


Round | states during the RTT-ELC grant period. Throughout the findings, it is important to keep 
the following caveats in mind: 


Rating levels do not reflect a uniform level of quality. Because states use different quality 
components and calculate rating levels differently, programs that receive a particular rating 
level might vary substantially in quality across states, even among states that use the same 
number of rating levels. In addition, within states, programs might achieve rating levels in 
different ways, weakening the link between the levels and certain measures of quality. For 
example, building block rating structures require programs to meet all standards within a 
level to receive a rating. In these structures, a program that missed qualifying for the next 
level based on a single standard might not be that different in quality from programs at the 
next level. In points and hybrid structures, programs that receive a given rating level could 
have met different standards, as long as they received enough points. 


Rating levels might not reflect programs’ “true” quality. States use various data 
collection and verification processes to help ensure the reliability of the rating levels, but 
ratings could differ from programs’ “true” quality for several reasons. First, some states use 
alternative pathways or automatic ratings, which exempt programs that meet external quality 
standards from part or all of the TQRIS rating process. It is not fully known whether these 
programs would meet the state-defined standards. Second, some states require programs to 
enter at the lowest level, instead of the level to which they would have qualified. In these 
states, ratings might initially understate some programs’ “true” quality. Overqualified 
programs could move up without improving quality, resulting in movement that does not 
reflect increased quality. Finally, programs could make quality improvements without 
applying for higher ratings; this could cause ratings to understate programs’ quality. 


The findings do not provide evidence that particular policies or program types caused 
the observed patterns in participation and ratings. Throughout the findings, we discuss 
the TQRIS characteristics and policies that might influence the patterns we present, but this 
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study is purely descriptive. The nine Round | states differ on many TQRIS characteristics 
and policies as well as on other dimensions, such as the characteristics of families seeking 
child care, the way in which TQRIS publicize the ratings to families, the composition of the 
early learning and development landscape, and the presence of other quality initiatives 
beyond TQRIS. Therefore, although we discuss several potential explanations for the 
patterns we present, other potential explanations might exist for these observed patterns. 
This brief does not describe all of the differences across the states that might have 
contributed to differences in the patterns of program participation or ratings that we present. 


Nonetheless, the findings from this brief could help provide a starting point for further 
discussion and research related to the TQRIS characteristics and policies that it describes. As the 
number of all types of programs participating in TQRIS continues to increase, it is of growing 
importance for states to consider how different TQRIS characteristics and policies may affect 
programs’ ratings, and, importantly, support an accurate rating of quality. 


Findings 


A. Research Question 1: What were the patterns in rating levels at TQRIS 
entry across states with different rating level policies? 


States use different policies to try to promote TQRIS entry among particular types of 
programs or ensure that programs meet initial requirements. However, these policies might 
channel programs toward ratings at entry that could differ from those they would have received 
on the TQRIS standards, potentially reducing the accuracy of programs’ ratings as measures of 
program quality. For example, alternative pathways and automatic ratings could fast-track state- 
funded pre-K and Head Start programs toward higher ratings, whereas entry-level requirements 
could push licensed centers to enter at the lowest rating level. 


These rating level policies might be a potential explanation for why state-funded pre-K and 
Head Start programs typically entered at one of the top two rating levels in RTT-ELC states, 
whereas licensed centers that did and did not receive subsides typically entered at one of the 
bottom two rating levels (Figure 2). Yet, if the patterns of high and low ratings for each program 
type persist even in states without the rating level policies, then these differences between ratings 
of each program type might simply reflect differences in whether these types of programs meet 
states’ standards. 
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Figure 2. Median TQRIS rating levels at entry, by program type and state, 2012-2016 


All states 


Minnesota 


North Carolina 


Ohio 


Washington 


California 


Rhode Island 
Delaware 
Maryland 
Massachusetts 
Lowest level Second lowest Middle Second highest Highest 
level level level level 


Median state-defined rating level at entry 


mw State-funded pre-K mHeadStart mLicensedcenters, subsidies mLicensed centers, no subsidies 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 9,658. Analysis included only programs that entered from 2012 to 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits data from California and Ohio before 2014: data were not available from California before 2014 and, due to significant 
system changes in Ohio in late 2013, the analysis included data only from 2014 to 2016. The median across states is calculated 
at the state level (rather than center level) so that states are weighted equally. 


Figure reads: In Massachusetts, the median entry rating level for state-funded pre-K programs, licensed centers receiving subsidies, and 
licensed centers not receiving subsidies was the lowest level. The median entry rating level for Head Start programs was the second lowest 
level. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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1. Patterns for state-funded pre-K and Head Start programs, by whether states offered 
them alternative pathways or automatic ratings 


Alternative pathways and alternative rating policies aim to exempt state-funded pre-K and 
Head Start programs from the burden of the full rating process, under the theory that they would 
likely meet state TQRIS standards. Similarly, high ratings for these programs, in both states with 
and without these policies, could suggest that the external standards for these programs and 
states’ TQRIS standards align well, potentially supporting the use of these policies. 


State-funded pre-K programs did not enter at higher rating levels in states with 
alternative pathways or automatic ratings than in states without these policies. About half 
of state-funded pre-K programs entered at the top two rating levels in both groups of states, 
though programs had lower ratings in states with alternative pathways or automatic ratings than 
in states without these policies (47 versus 56 percent; Figure 3, two bars closest to the right). The 
percentages of state-funded pre-K programs that entered at each rating level varied substantially 
across individual states, both among those with the rating level policies and those without. 


Among the four states that had an alternative pathway or automatic rating, two—Minnesota 
and Washington—had the majority of their state-funded pre-K programs enter at the highest or 
second highest rating level. Differences in how alternative pathways or automatic ratings are 
applied might help to explain why some state-funded programs in states with these policies did 
not enter the TQRIS in the highest rating levels. In Washington, the alternative pathway applied 
only to programs that met a threshold in the percentage of children (75 percent) served through 
state pre-K funds. Some state-funded pre-K programs might not have met this threshold and 
would not have been eligible for the alternative pathway. In Delaware, the alternative pathway 
did not apply to certain state-funded pre-K programs (those that operate in public schools), and 
these school-based programs typically entered the TQRIS at the lowest level. In contrast, in 
Maryland, the alternative pathway applied only to school-based state-funded pre-K programs, 
but none of these types of programs entered its TQRIS during the study period. 


Among the five states that did not have an alternative pathway or automatic rating, two— 
North Carolina and Ohio—had nearly all of their state-funded pre-K programs enter in the top two 
rating levels. This might reflect similarities between TQRIS standards and those of these states’ 
pre-K program in terms of how they define quality. TQRIS standards may not be well-aligned with 
state pre-K standards in the other three states, which had fewer state-funded pre-K programs 
entering at the top rating levels. 


States can also apply alternative pathways and automatic ratings to Head Start programs, 
which also tend to enter TQRIS in one of the top rating levels. Minnesota, Washington, and 
Delaware offered alternative pathways and automatic ratings to both state-funded pre-K and Head 
Start programs, but Maryland, Rhode Island, and Massachusetts offered them to only one of these 
program types. 


NCEE EVALUATION BRIEF 


16 QUALITY RATINGS AND SYSTEM CHARACTERISTICS 


Figure 3. Distribution of state-funded pre-K programs’ ratings at entry, by alternative pathways or 
automatic rating policy, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 2,669. Analysis included only programs that entered from 2012 to 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits data from California and Ohio before 2014: data were not available from California before 2014 and, due to significant 
system changes in Ohio in late 2013, the analysis included data only from 2014 to 2016. 


Figure reads: In Massachusetts, which did not have either an alternative pathway or automatic rating for its state-funded pre-K programs, 
57 percent of these programs entered at the lowest level, 25 percent at the second lowest level, O percent at the middle level, 18 percent 
at the second highest level, and O percent at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. Seven of the nine states 
required at least some types of programs to participate, which might have contributed to increased participation over time. For example, 
Rhode Island required programs that received Child Care and Development Fund subsidies to participate in TQRIS beginning in 2014. In that 
year, many licensed centers that received subsidies entered the system, increasing TQRIS participation by over 50 percentage points. 
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Head Start programs did not consistently enter at higher rating levels in states that 
offered them alternative pathways or automatic ratings than in states that did not. More than 
half of Head Start programs entered at the top two rating levels in both groups of states, with some 
differences in the average percentages across the two groups (Figure 4). States that had alternative 
pathways or automatic ratings had higher percentages of Head Start programs entering at the top 
two rating levels (65 versus 53 percent), but lower percentages entering at the top level (26 versus 
34 percent), than states without these policies. The percentages of Head Start programs that 
attained top rating levels varied substantially across individual states. Most Head Start programs 
entered in one of the top two rating levels in most states—four of the five states with these rating 
level policies and two of the four states without the policies. 


The different patterns in ratings at entry among states that had alternative pathways or 
automatic ratings might be related to variation in how Head Start programs could earn credit 
toward higher ratings. For example, in Minnesota, Head Start programs (similar to state-funded 
pre-K programs) were fast-tracked into the highest level by meeting a streamlined set of standards 
rather than going through the full TQRIS rating process. Programs had to demonstrate use of an 
approved curriculum and child assessment tool and deliver training to lead teachers on how to use 
them (Kirby et al. 2017). In contrast, Head Start programs in Massachusetts received credit toward 
several rating criteria but this credit did not meet the full requirements for any level, including the 
lowest one. Similarly, in Rhode Island, Head Start programs received a top rating on 3 of the 10 
standards in curriculum, assessment, and family communication and involvement, but this did not 
guarantee an overall rating at the highest levels. 


States that had automatic ratings or alternative pathways for both state-funded pre-K and Head 
Start programs also varied in how consistently they applied the policies to the different program 
types. For example, Delaware applied the automatic rating more uniformly to Head Start programs 
than state-funded pre-K programs. All of Delaware’s Head Start programs were eligible for an 
automatic rating at the state’s second highest level; 86 percent entered at that level and an 
additional 10 percent entered at the highest level. In contrast, Washington applied the same 
eligibility threshold (75 percent of children supported by a given type of fund) for an alternative 
pathway to both Head Start and state-funded pre-K programs. 


In states without alternative pathways or automatic ratings, the different patterns in the level 
of entry among Head Start programs might reflect differences in the alignment of TQRIS and Head 
Start standards. Similar to their state-funded pre-K programs, North Carolina and Ohio might have 
a higher degree of alignment between these standards than other states where fewer Head Start 
programs entered at high rating levels. 


Taken together, these findings suggest that alternative pathways or automatic ratings might 
not play a substantial role in channeling state-funded pre-K or Head Start programs toward high 
ratings. These policies are not consistently applied to programs within states. In addition, some 
states without these policies, such as North Carolina and Ohio, had nearly all of their state-funded 
pre-K and Head Start programs enter in top rating levels, suggesting that, at least in some states, 
standards for these programs and TQRIS might be relatively well aligned. 
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Figure 4. Distribution of Head Start programs’ ratings at entry, by alternative pathways or automatic 
rating policy, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 935. Analysis included only programs that entered from 2012 to 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as 
having no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are 
statewide. The figure omits data from California and Ohio before 2014: data were not available from California before 2014 and, 
due to significant system changes in Ohio in late 2013, the analysis included data only from 2014 to 2016. 


Figure reads: In Maryland, which did not have either an alternative pathway or automatic rating for its Head Start programs, 84 percent of 
its programs entered at the lowest level, 11 percent entered at the second lowest level, 4 percent entered at the middle level, O percent 
entered at the second highest rating level, and 1 percent entered at the highest rating level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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2. Patterns for licensed centers that did and did not receive subsidies, by whether states 
required these programs to enter at the lowest rating level 


A separate rating level policy—trequired entry at the lowest rating level—could push both 
types of licensed centers toward the lowest rating level at entry. Low ratings for these programs, 
both in states with and without these requirements, could suggest that these types of licensed 
centers are not meeting state TQRIS standards and have ample room to improve. 


Similar percentages of licensed centers that received subsidies entered at the lowest 
rating level in states that required them to do so as in states that did not. There was little 
difference between states that had an entry level requirement and those that did not in the 
percentages of licensed centers that received subsidies entering at the lowest level (50 versus 47 
percent, Figure 5, bar furthest to the left). 


In two of the four states with the entry level requirements—Washington and Massachusetts— 
a majority of licensed centers that received subsidies entered at the lowest rating level. Differences 
in implementation could help to explain why some states with this requirement had fewer programs 
enter at the lowest rating level than might be expected. For example, although Delaware and North 
Carolina required licensed centers that received subsidies to enter the TQRIS at the lowest level, 
the first recorded ratings were often not at the lowest level, suggesting that movement out of the 
lowest level might have happened quickly. These two states did not require programs to move 
through the rating levels sequentially, so programs could advance to higher ratings quickly. North 
Carolina also had another policy in place that might have counteracted the entry level requirement 
for centers that received subsidies; licensed centers that wanted to receive subsidies were required 
to attain North Carolina’s middle-level rating to be eligible for the subsidy. In fact, 50 percent of 
licensed centers that received subsidies in North Carolina had first recorded ratings at the middle 
level (and thus appear to enter at that level). 


States without these entry level requirements also frequently had high percentages of licensed 
centers that received subsidies entering at the lowest rating level. For example, in three of the five 
states that did not require these programs to enter at the lowest level—Maryland, Rhode Island, 
and Ohio—a majority of them entered there anyway. The low ratings for licensed centers that 
received subsidies on these states’ TQRIS standards suggests that these types of programs might 
have substantial room for improvement. Minnesota, which did not have this requirement, was the 
only state in which more than half of licensed centers that received subsidies entered the TQRIS 
at the two highest levels. 
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Figure 5. Distribution of ratings at entry for licensed centers that received subsidies, by entry 
requirement at lowest level, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 4,421. Analysis included only programs that entered from 2012 to 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits data from California and Ohio before 2014: data were not available from California before 2014 and, due to significant 
system changes in Ohio in late 2013, the analysis included data only from 2014 to 2016. 


Figure reads: In Minnesota, which did not require licensed centers that received subsidies to enter at the lowest rating level, 7 percent of 
these centers entered at the lowest level, 18 percent at the second lowest level, O percent at the middle level, 5 percent at the second 
highest level, and 70 percent at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Higher percentages of licensed centers that did not receive subsidies entered at the lowest 
rating level in states that required them to do so than in states that did not. States with the 
entry requirement had higher percentages of licensed centers that did not receive subsidies entering 
at the lowest rating level than states without this requirement (56 versus 38 percent, Figure 6). 


In states with an entry requirement, the percentages of this program type that entered at the 
lowest rating level ranged from 96 percent in Washington to 4 percent in Delaware. In North 
Carolina and Delaware, similar to licensed centers that received subsidies, licensed centers that 
did not receive subsidies might have moved out of the lowest level quickly. 


In states that did not have an entry requirement, the percentage of licensed centers that did not 
receive subsidies that entered at the lowest level ranged from 89 percent in Maryland to zero in 
Minnesota. In Rhode Island, Ohio, and Minnesota, licensed centers that did not receive subsidies 
actually entered at higher rating levels than those that received subsidies, which might help drive 
the differences in findings across the two types of programs. 


Thus, requiring entry at the lowest rating level had some association with the entry ratings of 
licensed centers that did not receive subsidies but no association for licensed centers that received 
subsidies. This could suggest the need to examine whether entry level requirements hold licensed 
centers, particularly those that do not receive subsidies, at lower rating levels than they might have 
received based on the standards. 


The differences across rating levels at entry for the four types of programs, combined with 
differences across states in the composition of programs that participate in TQRIS, might 
contribute, in part, to differences in patterns of ratings across states. For example, in California 
and Minnesota, state-funded pre-K programs—which tended to receive high ratings—comprise 
the majority of programs that participate in TQRIS (Appendix B, Figure B.1). In contrast, in the 
other seven states, licensed centers that receive subsidies—which tend to receive lower ratings— 
are in the majority. The next section focuses on several other key TQRIS characteristics and 
policies that could partially influence differences in ratings across states. 
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Figure 6. Distribution of ratings at entry for licensed centers that did not receive subsidies, by entry 
requirement at lowest level, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 1,633. Analysis included only programs that entered from 2012 to 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits data from California and Ohio before 2014: data were not available from California before 2014 and, due to significant 
system changes in Ohio in late 2013, the analysis included data only from 2014 to 2016. 


Figure reads: In Minnesota, which did not have an entry level requirement for licensed centers that did not receive subsidies, 25 percent of 
programs entered at the second lowest level, compared with 75 percent for the highest rating level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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B. Research Question 2: What were the patterns in programs’ attainment of 
the top two rating levels by 2016 across states with different TQRIS 
characteristics and policies? 


The percentages of programs that attained the top two rating levels by 2016 varied 
dramatically across states (Figure 7). Two states—Minnesota and North Carolina—had more than 
75 percent of programs at the top rating levels. Minnesota had the majority of its programs 
receiving these ratings at entry through alternative pathways, and North Carolina, which had the 
longest-running TQRIS, had most of its programs achieving these rating levels before the 
beginning of the study period. In contrast, the four states with less than 25 percent of programs 
achieving the top rating levels—Rhode Island, Massachusetts, Washington, and Maryland—had 
either newer TQRIS or block rating structures. 


Thus, states differed on multiple TQRIS characteristics and policies, as well as other 
dimensions that could be associated with these differences in programs’ ratings. Key differences 
include the length of TQRIS implementation, rating structure, and offer of tiered subsidy 
reimbursement rates. Differences in ratings between states that have each of these characteristics 
or policies and those that do not could provide evidence about the potential relationships between 
these characteristics or policies and programs’ ratings. 


To examine differences in ratings, we focus on programs attaining the top two rating levels 
by 2016, the end of the RTT-ELC grant period. However, findings are similar for the full 
distribution of ratings by that year. (See Appendix B, Figures B.2 to B.4, for the ratings 
distributions for the two characteristics and one policy examined.) 
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Figure 7. Percentages of programs that achieved the highest or second highest state-defined TQRIS 
rating level in 2016 and how they got there, by state 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis was based on all programs that achieved the highest or second highest rating 
level in each state in 2016. “Achieved by start of analysis period” includes programs that were at the highest or second highest 
rating at the beginning of the study period (in 2014 for California and Ohio and in 2012 for the other seven states) and for which 
we cannot determine whether they had achieved those levels at entry. California data are from 16 counties that implemented 
TQRIS beginning in 2012; all other data are statewide. 


Figure reads: Twenty-two percent of all programs that participated in TQRIS in 2016 in Rhode Island were at the highest or second-highest 
rating level; 3 percent of programs received the highest or second highest rating by the start of the analysis period, 7 percent of programs 
were not eligible for an automatic or alternate pathway and received the highest or second highest rating level at entry, 5 percent were 
eligible for one of these policies and received the highest or second highest rating level at entry, and an additional 7 percent achieved the 
highest level through upward movement. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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1. Patterns for all programs, by length of implementation 


At the start of the RTT-ELC grants, some states—California, Maryland, Minnesota,* and 
Washington—were just launching their TQRIS, but other states had implemented their TQRIS for 
longer. For example, North Carolina had the longest running TQRIS, which had been in place 
since 1999, followed by Ohio, which implemented its TQRIS in 2006. 


In states that had implemented TQRIS for longer, programs might have had more time to 
improve, and states might have had more time to develop and implement other policies that aim to 
support program improvement. As a result, programs in states with more established TQRIS might 
have had higher ratings by the end of the grant period. 


States that had implemented statewide TQRIS for 10 or more years by 2016 had a higher 
percentage of programs in the top two rating levels than other states. In the two states with 
long-established statewide TQRIS of 10 or more years by 2016—North Carolina and Ohio—more 
than 50 percent of programs achieved one of the top two rating levels (Figure 8). States with 
TQRIS that had operated statewide for fewer than 10 years by 2016 had lower percentages of 
programs reaching the top levels, but these percentages varied substantially across states. 


Three states that had implemented TQRIS for fewer than 10 years by 2016—Minnesota, 
California, and Delaware—had similar or higher percentages of top rated programs as North 
Carolina and Ohio. This could be due, in part, to the percentages of programs in these states that 
attained these ratings at entry. In Minnesota, 80 percent of programs that were at the highest state- 
defined levels in 2016 achieved those levels at entry and were eligible for an alternative pathway. 
In California and Delaware, a majority of programs had achieved one of the top rating levels by 
2016 through a combination of high ratings at entry or ratings improvement that was not related 
to an alternative pathway or automatic rating. Minnesota and Delaware had also begun their 
TQRIS pilots as early as 2007, so programs from the pilot counties might have had nearly 10 years 
to improve their ratings. However, of these two states, only Delaware had a substantial percentage 
of its programs at the top rating levels that had achieved these ratings at the start of the study period 
or through movement up. 
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rating level in 2016 and how they got there, by length of TQRIS implementation 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis was based on all programs that achieved the highest or second highest 
rating level in each state in 2016. “Achieved by start of analysis period” includes programs that we first observed at the highest 
or second highest rating (in 2014 for California and Ohio and in 2012 for the other seven states) and for which we cannot 
determine whether they had achieved those levels at entry. California data are from 16 counties that implemented TQRIS 
beginning in 2012; all other data are statewide. The averages across States in the first three bars are calculated at the state level 
(rather than center level) so that states are weighted equally for each average. 


Figure reads: In North Carolina, which implemented TQRIS 10 or more years ago, 43 percent of programs achieved one of the top two 
rating levels by start of analysis period, 17 percent achieved one of the top two ratings at entry and were not eligible for automatic rating or 
alternative pathway, O percent achieved one of the top two ratings at entry and were eligible for automatic rating or alternative pathway, 
and 18 percent achieved one of the top two ratings through movement up. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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2. Patterns for all programs, by rating structure 


States use one of three rating structures to determine programs’ ratings: building block, points, 
or hybrid. Points and hybrid rating structures might promote more movement than building block 
structures because they provide programs different avenues to meet the standards for each rating 
level. For example, they might allow a program to earn enough total points for a high rating by 
earning many points on one standard and few on another (such as earning more points toward 
curriculum and assessment standards and fewer toward staff qualifications standards). In contrast, 
building block structures require programs to meet minimums for each standard. A program rated 
using a building block structure might not receive a top rating level because it fails to meet just 
one standard, even if it meets or exceeds the minimums for all of the other standards. Simulation 
studies have found that building block rating structures can hold programs at lower rating levels, 
whereas the same programs could qualify for higher levels under a points or hybrid structure (Isner 
et al. 2012; Tout et al. 2014). 


Substantially lower percentages of programs achieved top rating levels in states with 
building block rating structures than in states with hybrid or points rating structures. On 
average, less than 20 percent of programs attained top rating levels by 2016 in states with a building 
block rating structure, compared with more than 50 percent for points or hybrid rating structures 
(Figure 9). 


In all three states that had a building block rating structure—Maryland, Massachusetts, and 
Rhode Island—fewer than one-quarter of programs had achieved one of the top two rating levels 
by 2016. About 7 percent of programs in each of the three states achieved these levels through 
movement up. 


The four states with hybrid rating structures varied widely in the percentage of programs that 
achieved one of the top two state-defined rating levels in 2016—from 10 percent in Washington 
to 95 percent in Minnesota. The ways in which programs achieved the top levels in each of these 
four states also varied. Both Delaware and Ohio had more than 20 percent of programs that 
achieved one of the two highest rating levels by 2016 through ratings improvement. The vast 
majority of programs in Minnesota achieved these rating levels at entry through alternative 
pathways. 


Both states with points rating structures—California and North Carolina—had a high 
percentage of top-rated programs in 2016 (60 and 78 percent, respectively). Both states had 
substantial percentages of programs that achieved top rating levels through movement up the 
ratings. 


These findings are consistent with the hypothesis that points and hybrid rating structures could 
promote more movement than building block rating structures. However, there are other potential 
explanations for these findings, including the fact that the three states with building block rating 
structures were relatively early in their implementation of TQRIS. Massachusetts, which began its 
TQRIS in 2011, did not begin granting the second highest and highest rating levels until 2014. 
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Figure 9. Percentages of programs that achieved the highest or second highest state-defined TQRIS 
rating level in 2016 and how they got there, by rating structure 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis was based on all programs that achieved the highest or second highest rating 
level in each state in 2016. “Achieved by start of analysis period” includes programs that we first observed at the highest or second 
highest rating (in 2014 for California and Ohio and in 2012 for the other seven states) and for which we cannot determine whether 
they had achieved those levels at entry. California data are from 16 counties that implemented TQRIS beginning in 2012; all other 
data are statewide. The averages across states in the first three bars are calculated at the state level (rather than center level) so 
that states are weighted equally for each average. 

Figure reads: In North Carolina, which uses a points rating structure, 43 percent of programs achieved one of the top two rating levels by 

start of analysis period, 17 percent achieved one of the top two ratings at entry and were not eligible for automatic rating or alternative 

pathway, O percent achieved one of the top two ratings at entry and were eligible for automatic rating or alternative pathway, and 18 percent 
achieved one of the top two ratings through movement up. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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3. Patterns for licensed centers that received subsidies, by whether states offered tiered 
reimbursement rates 


Seven of the nine RTT-ELC Round | states offer a tiered reimbursement rate to supplement 
the rate at which public funding covered the costs of providing care to children from low-income 
households. These states offer a higher subsidy rate for programs that achieve a minimum TQRIS 
rating, or require a minimum rating, typically the middle level, to qualify for the subsidy at all. 
The states then offer an increasingly higher subsidy rate—or tiered rate—as centers move up to 
higher state-defined TQRIS levels above the minimum required. States can structure their tiered 
reimbursement rates differently to encourage a variety of outcomes, such as encouraging programs 
to participate in a TQRIS, serve children from low-income households, or improve quality. These 
tiered rates could potentially motivate movement up the TQRIS rating levels among licensed 
centers that receive these subsidies. 


Tiered reimbursement rates were associated with higher percentages of licensed centers 
that received subsidies achieving top rating levels by 2016, but states with tiered 
reimbursement rates had higher ratings for all types of programs, even those that did not 
receive this financial incentive. On average, 39 percent of licensed centers that received subsidies 
achieved top two rating levels in states that offered a tiered retmbursement rate, compared with 20 
percent for other states (Figure 10). 


Patterns of achieving the top two rating levels varied widely in states that offered tiered 
reimbursement rates and those that did not. Among states that offered tiered reimbursement rates, 
the percentage ranged from 4 percent in Washington to 84 percent in Minnesota. The two states 
that did not offer tiered reimbursement rates—California and Massachusetts—had 35 and 5 
percent of licensed centers that received subsidies achieving top rating levels by 2016. 


However, states that offered tiered reimbursement rates had higher ratings for all types of 
programs, even those that did not receive this financial incentive. For example, licensed centers 
that did not receive subsidies also had higher median ratings in states with tiered reimbursement 
rates than in other states (Figure 11). Thus, variations in states’ many other different TQRIS 
characteristics and policies might mask associations between this financial incentive and ratings. 
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Figure 10. Percentages of licensed centers receiving subsidies that achieved the highest or second- 
highest state-defined TQRIS rating level in 2016 and how they got there, by offer of tiered 
reimbursement rates 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 6,037. Analysis was based on all programs that achieved the highest or second highest rating 
level in each state in 2016. “Achieved by start of analysis period” includes programs that we first observed at the highest or 
second highest rating (in 2014 for California and Ohio and in 2012 for the other seven states) and for which we cannot 
determine whether they had achieved those levels at entry. California data are from 16 counties that implemented TQRIS 
beginning in 2012; all other data are statewide. The averages across states in the first two bars are calculated at the state level 
(rather than center level) so that states are weighted equally for each average. Because licensed centers that receive subsidies 
are almost always ineligible for alternative pathways or automatic ratings, this figure does not include a category for “achieved at 
entry, eligible for automatic rating or alternative pathway.” 


Figure reads: |n California, which did not offer tiered reimbursement rates to licensed centers that received subsidies, 5 percent of 
licensed centers that received subsidies achieved one of the top two rating levels by start of analysis period, 14 percent achieved one of 
the top two ratings at entry and were not eligible for automatic rating or alternative pathway and 17 percent achieved one of the top two 
ratings through movement up. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure 11. Median TQRIS rating levels for each program type, by offer of tiered reimbursement rates, 
2016 


Averages across all states 


All states 


States that offered tiered 
reimbursement rates 
States that did not offer 

tiered reimbursement rates 


States that offered tiered reimbursement rates to licensed centers that received subsidies 


Minnesota 
Delaware 
North Carolina 
Ohio 

Rhode Island 


Maryland 


Washington 


States that did not offer tiered reimbursement rates to licensed centers that received susbidies 


California 


Massachusetts 


Lowestlevel Second lowest Middle Secondhighest Highest 
level level level level 


Median state-defined rating level at entry 


= Licensed centers, subsidies Licensed centers, nosubsidies mHead Start m# State-funded pre-K 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis included only programs that were rated. Rating categories reflect the order of 
ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 
Massachusetts started granting the second highest and highest rating levels in January and November 2014, respectively. 
California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The median across 
states is calculated at the state level (rather than center level) so that states are weighted equally. 


Figure reads: The median rating level for programs rated by TQRIS in 2016 in Massachusetts was the lowest level for licensed center 
receiving subsidies and licensed center not receiving subsidies and the second lowest level from Head Start programs and state-funded 
pre-K programs. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Conclusion 


RTT-ELC tasked states with developing TQRIS that promote participation among a range of 
early learning and development programs. It also encouraged states to increase the number of 
programs that attain the highest rating levels, particularly to increase access to high quality 
programs for high-needs children. TQRIS have developed in different ways, with each state 
selecting a set of TQRIS characteristics that are unique and largely untested. The relative newness 
of many TQRIS presents challenges to administrators and policymakers interested in making 
informed decisions about how to structure or refine their systems to achieve their goals. This 
analysis takes a step toward understanding programs’ ratings at TQRIS entry and achievement of 
the highest rating levels by describing patterns in states that have made different decisions about 
how to structure the TQRIS or make use of certain policies. 


Cross-state analyses of these systems can provide important information about trends that are 
worth exploring further. The descriptive patterns presented in this brief reinforce the disparities 
across states in how TQRIS are structured and the policies used within them. However, the findings 
do not provide evidence that particular policies caused the observed patterns in programs’ ratings. 
Each TQRIS is a unique constellation of characteristics (such as rating structure) and policy 
decisions (such as alternative pathways) that combine with the composition of the standards that 
comprise the ratings themselves and the longevity of implementation to make a complex system. 
Disentangling the pieces to identify what works and for which programs is challenging, yet states 
desire information to structure and refine TQRIS in ways that will meet the goals of quality 
improvement for programs and good outcomes for children. 


The patterns identified by this descriptive work suggest several issues that future research 
could investigate. First, in the nine Round | RTT-ELC states, alternative pathways and automatic 
ratings were generally not related to higher ratings at entry for state-funded pre-K or Head Start 
programs. This appeared to be due, in part, to inconsistencies in how states applied alternative 
pathway and automatic ratings policies, as well as high ratings among states without these policies. 
The latter could suggest that, at least in some states, external standards for these types of programs 
could align well with states’ TQRIS standards. Thus, it might be useful for research to explicitly 
test the alignment across sets of standards. For example, future studies could try to create a 
crosswalk between the sets of standards or conduct the full rating process on a sample of the 
programs that would receive alternative pathways or automatic ratings to verify the accuracy of 
their ratings. 


Second, requiring programs to enter at the lowest rating level was associated with lower 
ratings at entry for licensed centers that did not receive subsidies but not for those that received 
subsidies. Entry level requirements, along with other factors, might play some role in whether 
these programs enter at lower levels. However, it is also possible that licensed centers realistically 
have substantial room to improve their ratings. Future research might further examine how entry 
requirements affect the accuracy of programs’ rating at entry. 


Third, states that had implemented TQRIS for more than 10 years by 2016 had higher 
percentages of programs at top rating levels. This could reflect differences in the TQRIS 
characteristics and policies implemented by states that had implemented TQRIS the longest, 
programs in these states having more time to reach the top rating levels, or other differences 
between these states and states that adopted TQRIS more recently. As TQRIS continue to evolve, 
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it will be important to understand how long it takes for states to fully implement systems and 
policies that might promote programs’ improvement and attainment of the top rating levels. 


Fourth, states with building block rating structures had lower percentages of programs that 
attained top rating levels than states with points or hybrid rating structures. This finding does not 
necessarily imply that particular rating structures cause higher ratings. For example, states that had 
building block rating structures also differed in other ways from states that had points or hybrid 
rating structures, such as in their length of TQRIS implementation. Nevertheless, the patterns of 
ratings for states with different rating structures are in line with prior research that found that 
building block rating structures might hold programs at lower rating levels by requiring them to 
meet all of the standards, in contrast to point or hybrid rating structures, which could offer 
programs some flexibility to choose which standards they meet. Yet, these differences in how the 
rating structures treat standards might result in very different definitions of quality for a given 
rating. This suggests the need for additional research on how quality varies across programs that 
attain the same rating within each rating structure. In addition, little is known about which rating 
structures (and definitions of quality) lead to better outcomes for children. Across various states 
(and rating structures), previous research has found little consistent evidence of an association 
between programs’ TQRIS ratings and children’s outcomes (Sabol et al. 2013; Tout et al. 2017). 


Fifth, patterns of ratings for licensed centers that received subsidies provided limited 
information about the relationship between their ratings and states’ offer of tiered subsidy 
reimbursement rates. Additional research is needed to test the effectiveness of financial incentives 
and other supports that could help these programs improve. 


Finally, this brief does not examine how states’ TQRIS characteristics or policies relate to 
programs’ participation in TQRIS, and it focuses on only nine states, each of which measure 
quality differently. Additional research that examines how TQRIS characteristics and policies 
affect programs’ participation and quality, using consistent measures of the child care environment 
or children’s development across states and program types could be useful. Such research could 
further inform states’ decisions about TQRIS and help facilitate increased access to high quality 
early care and education experiences for children. 
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Endnotes 


' For more information on the history and implementation of TQRIS, see the first report from 
this study, Kirby et al. (2017). 


? Information from BUILD Initiative (2017) on the current 44 TQRIS indicates a wide range 
in characteristics such as rating structure and validity period. For example, 18 TQRIS use a hybrid 
rating structure, 19 a building block structure, and 7 a points structure. The validity period is three 
years in 17 TQRIS, two years in 9, one year in 8, and a period that falls outside these categories in 
10 TQRIS. 


3 Rhode Island did not call its incentives tiered reimbursement but they functioned in the same 
way and we classified the policy as such for cross-state aggregation. 


4 We considered Minnesota as implementing its TQRIS statewide in 2012, the year that 


programs in 22 of the state’s 87 counties were eligible to participate in the full-rating process; 
eligibility gradually expanded to all counties by 2015. 
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APPENDIX A. METHODS AND SAMPLE SIZES 


This appendix provides additional details on data sources, standardizing variables across 
states, sample sizes, and defining program types. To support cross-state aggregation, we used 
information from administrative data sources from multiple states to classify programs into 


types. 


State administrative data sources. The administrative data we collected came from 
multiple sources within each state. The landscape of center-based early learning and 
development programs that serve young children is quite diverse, and states have typically 
decentralized program oversight. For example, often distinct offices with separate data systems 
administer licensing and child care subsidies but both are generally under the auspices of a 
state’s human or social services agency; state-funded pre-K programs are generally under the 
state’s department of education; and Head Start is a federally administered program. The RTT- 
ELC grants aimed, in part, to support the integration of state early care and education systems, 
but the data still reside in many different agencies and entities. As a result, the data we requested 
came from multiple sources within each state. In five states (Maryland, Massachusetts, 
Minnesota, North Carolina, and Washington), we worked with a main contact in one department 
who coordinated with others in the state to gather and submit the data. In the four other states 
(California, Delaware, Ohio, and Rhode Island), we obtained data from a variety of sources. 


Standardizing variables across states. We used information from multiple sources to 
maximize our ability to determine licensing status, funding sources, and program closings and to 
build a longitudinal analysis file for each state. To analyze results across states, we used a 
consistent method to construct variables from the raw data each state provided. Our definitions 
of program types or other program elements may not exactly match each state’s definition. As a 
result, the numbers in this brief may differ from those in the annual progress reports that states 
submitted to ED for RTT-ELC. The data from the state RTT-ELC yearly reports are difficult to 
assess across states, given the differences in data coverage of children and programs, the 
methods of classifying programs by type, and the methods of counting programs in rating levels. 
In addition, because those data represent a single point in time (that is, they are not longitudinal), 
discerning patterns of TQRIS entry and movement up the rating levels by specific program types 
over time is challenging. 


Sample sizes. We collected information about all early learning and development programs 
in the state—both those that participated in the TQRIS and those that did not—to provide a 
complete picture of how availability to high quality programs may have changed under RTT- 
ELC. The number of early learning and development programs varies across the states and, 
therefore, influences the number of programs that can participate in the TQRIS. Some states 
count programs as participating in TQRIS before they officially receive a rating. For analyses on 
ratings in this brief, we included only programs that had received a TQRIS rating. Table A.1 
presents the numbers of all early learning and development programs, the number of TQRIS 
participating programs, and the number of TQRIS-rated programs for each year in the analysis 
period by state. 
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Table A.1. Total number of early learning and development programs in the nine Round 1 RTT-ELC 
states and number of TQRIS participating and rated programs, 2012 to 2016 


State and number of programs 


California 
Total na. n.a. 7,616 8,406 8,442 
Participating n.a. na. 1,214 2,009 2,226 
Rated na. n.a. 949 1,913 2,153 
Delaware 
Total 373 426 452 455 451 
Participating 123 253 362 378 382 
Rated 123 253 362 377 379 
Maryland 
Total 1,947 1,934 1,922 1,920 1,919 
Participating 0 102 447 1,011 1,266 
Rated (0) 102 447 1,002 1,257 
Massachusetts 
Total 2,384 2,713 2,679 2,795 2,597 
Participating 814 1,019 1,103 1,215 1,175 
Rated 795 986 1,072 1,166 1,125 
Minnesota 
Total 2,851 2,851 2,851 2,851 2,851 
Participating 485 1,312 1,586 1,836 1,844 
Rated 485 1,312 1,584 1,827 1,831 
North Carolina 
Total 4,351 4,401 4,355 4,300 4,116 
Participating 3,713 3,756 3,699 3,738 3,629 
Rated 3,651 3,699 3,652 3,696 3,594 
Ohio 
Total n.a. n.a. 6,085 6,024 5,813 
Participating na. na. 14144 1561 1,739 
Rated na. n.a. 1,128 1,534 1,722 
Rhode Island 
Total 326 332 334 335 330 
Participating 40 55 237 265 264 
Rated 40 55 237 265 261 
Washington 
Total 2,055 1,994 1,932 1,957 1,836 
Participating 356 815 967 1,035 1,089 
Rated 320 773 924 987 1,035 
All states except California and Ohio 
Total 14,287 14,651 14,525 14,613 14,100 
Participating 5,531 7,312 8,401 9,478 9,649 
Rated 5,414 7,180 8,278 9,320 9,482 
All states including California and Ohio 
Total na. n.a. 28,226 29,043 28,355 
Participating na. na. 10,756 13,048 13,614 
Rated na. n.a. 10,355 12,767 13,357 
Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 
Note: Analysis counted all licensed centers, Head Start, and state-funded pre-K programs in each state. A program was 


considered participating if it was in the TQRIS at any point during the year. Rated programs were analyzed as of 
June of each year; therefore, numbers of rated programs may differ slightly than number of participating 
programs even for states that do not generally have unrated programs. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. 


The table omits California and Ohio before 2014. Data were not available from California before 2014. Due to 
significant system changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. Data for 
2016 are from January to June. 


n.a. = not applicable; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement 
system. 
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Defining program types. We developed mutually exclusive program types to mirror those 
typically found in the state RTT-ELC yearly reports as follows: 


e State-funded pre-K.* Includes (1) programs in a school-based setting that received state 
pre-K funds; (2) programs that received only state pre-K funds, or a combination of state 
pre-K and Child Care and Development Fund or other child care subsidy funds, but not 
Head Start; and (3) a small percentage of licensed centers that received both Head Start and 
state pre-K funds but did not have Head Start in the program name. 


e Head Start. Includes (1) programs that had Head Start in their name, but also might have 
received Child Care and Development Fund or other child care subsidy funds, or state pre-K 
funding; and (2) programs that did not have Head Start in their name, but received only 
Head Start funding, or that received Head Start and Child Care and Development Fund or 
other child care subsidy funds. 


e Licensed centers, received subsidies. Includes licensed centers that received Child Care 
and Development Fund or similar subsidies to support low-income children in the program 
but did not receive Head Start or state pre-K funding. 


e Licensed centers, no subsidies. Includes licensed centers that did not receive Child Care 
and Development Fund or other child care subsidy, Head Start, or state pre-K funding. 


Table A.2 presents the percentage of programs within each program type as well as the 
percentages, by program type, that were licensed, and that received the various sources of public 
funding. 


All analyses exclude license-exempt programs that were not either state-funded pre-K or 
Head Start programs (which are often considered license-exempt when operated in public 
schools). Most states did not provide data on these types of license-exempt programs. Only three 
states allowed license-exempt programs that were not receiving state pre-K or Head Start funds 
to participate in TQRIS. Such programs could include those administered on Tribal lands or on 
military bases, or those affiliated with a religious program. These programs comprised less than 
1 percent of TQRIS participating programs in 2016 in two of the three states and less than 5 
percent in the third state. 


“ We classified state-funded Head Start in Delaware as state-funded pre-K to ensure program types across the 
nine states were comparable and to distinguish the federal Head Start program from state-funded programs. 
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Table A.2. Licensing status and funding of TQRIS participating programs in 2016, by program type 


Percentage of programs receiving each 
type of funding 


Funding 
from 
Number of Percentage of Percentage Childcare Head Start State pre-K multiple 
Program type programs programs licensed subsidies funds funds sources 


All programs participating in TQRIS 13,614 100 87 57 13 31 13 
State-funded pre-K 3,920 29 60 28 6 100 31 

ria 2,174 16 28 5 2 100 6 

Licensed pre-K 1,746 13 100 55 12 100 61 
Head Start 1,468 44 90 26 100 19 40 
Licensed, received subsidies 6,325 46 100 100 0 0 0 
Licensed, no subsidies 1,901 14 100 0 0 0 0 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: This analysis included all state-funded pre-K, Head Start, and licensed centers that each state considered participating in 2016, 
including those that had not yet received a rating. School-based pre-K includes a small number of license-exempt programs 
receiving state pre-K funding. The analysis excluded all other license-exempt programs. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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APPENDIX B. ADDITIONAL FINDINGS 


This appendix presents additional findings. Figure B.1 presents the composition of TQRIS 
participating programs by state. Figures B.2 through B.4 present the distribution of ratings in 
each state, by length of TQRIS implementation, rating structure, and offer of tiered 
reimbursement rates. 


Figure B.1. Composition of TQRIS participating programs, by state, 2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of TQRIS-rated programs was 13,357 in 2016. Analysis included licensed centers, Head Start programs, and 
state-funded pre-K programs that were considered participating by each state, including those that had not yet received a rating. 
California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. 


Figure reads: Of the programs that were rated by TQRIS in 2016 in California, 8 percent were licensed centers that received subsidies, 19 
percent were licensed centers that did not receive subsidies, 21 percent were Head Start programs, and 52 percent were state-funded 
pre-K programs. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.2. Percentages of programs in each state-defined TQRIS rating level in 2016, by length of 
TQRIS implementation 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis included only rated programs. Rating categories reflect the order of ratings 
within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 
Massachusetts started granting the second highest and highest rating levels in January and November 2014, respectively. 
California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The average 
across states is calculated at the state level (rather than center level) so that states are weighted equally. 


Figure reads: In North Carolina, which implemented TQRIS 10 or more years ago, 2 percent of programs were rated at the lowest level, 1 
percent were rated at the second lowest level, 20 percent were rated at the middle level, 26 percent were rated at the second highest 
level, and 52 percent were rated at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.3. Percentages of programs in each state-defined TQRIS rating level in 2016, by rating 
structure 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis included only rated programs. Rating categories reflect the order of ratings 
within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 
Massachusetts started granting the second highest and highest rating levels in January and November 2014, respectively. 
California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The average 
across states is calculated at the state level (rather than center level) so that states are weighted equally. 


Figure reads: In North Carolina, which used a points rating structure, 2 percent of programs were rated at the lowest level, 1 percent were 
rated at the second lowest level, 20 percent were rated at the middle level, 26 percent were rated at the second highest level, and 52 
percent were rated at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.4. Percentages of licensed programs receiving subsides in each state-defined TQRIS rating 
level in 2016, by offer of tiered reimbursement rates 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis included only programs that were rated. Rating categories reflect the order of 
ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 
Massachusetts started granting the second highest and highest rating levels in January and November 2014, respectively. 
California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. 


Figure reads: In California, which did not offer tiered reimbursement rates to licensed centers that received subsides, 11 percent of 
programs were rated at the lowest level, 25 percent were rated at the second lowest level, 29 percent were rated at the middle level, 33 
percent were rated at the second highest level, and 3 percent were rated at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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For more information on the full study, please visit: 


https://ies.ed.gov/ncee/projects/evaluation/other racetotop.as 
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